1. Field of the Invention
This invention is related in general to the field of microscopy. In particular, it relates to a novel approach for acquiring multiple image swaths of a large sample area using an array microscope and subsequently combining the swaths to form a good-quality high-resolution composite image.
2. Description of the Related Art
Typical microscope objectives suffer from the inherent limitation of only being capable of imaging either a relatively large area with low resolution or, conversely, a small area with high resolution. Therefore, imaging large areas with high resolution is problematic in conventional microscopy and this limitation has been particularly significant in the field of biological microscopy, where relatively large samples (in the order of 20 mm×50 mm, for example) need to be imaged with very high resolution. Multi-element lenses with a large field of view and a high numerical aperture are available in the field of lithography, but their cost is prohibitive and their use is impractical for biological applications because of the bulk and weight associated with such lenses.
A recent innovation in the field of light microscopy provides a solution to this problem using a new type of array microscope. As described in commonly owned U.S. Pat. No. 7,184,610, herein incorporated by reference, this new array microscope consists of an array of miniaturized microscopes wherein each includes a plurality of optical elements individually positioned with respect to a corresponding image plane and configured to image respective sections of the sample object. As illustrated in FIG. 1, the array further includes a plurality of image sensors corresponding to respective optical elements and configured to capture image signals from respective portions of the object. The absolute magnification in an array microscope is greater than one, which means that it is not possible to image the entire object surface at once even when it is equal to or smaller than the size of the array. Rather, the imaged portions of the object are necessarily interspaced in checkerboard fashion with parts of the object that are not imaged. Accordingly, this array microscope was designed in conjunction with the concept of linear object scanning, where the object is moved relative to the array microscope and data are acquired continuously from a collection of linear detectors. Data swaths obtained from individual optical systems are then concatenated to form the composite image of the object.
In such an array microscope, a linear array of miniaturized microscopes is preferably provided with adjacent fields of view that span across a first dimension of the object and the object is translated past the fields of view across a second dimension to image the entire object. As illustrated in FIG. 2, because each miniaturized microscope 10 is larger than its field of view 12 (having, for example, respective diameters of about 1.8 mm and 200 μm—only one is shown for simplicity), the individual microscopes of the imaging array are staggered in the direction of scanning (x direction in the figure) so that their relatively smaller fields of view are offset over the second dimension but aligned over the first dimension. As a result of such staggered arrangement of the rows of miniaturized microscopes, the continuous swaths covered by the linear scan of each optical system is substantially free of overlap with the continuous swaths covered by adjacent optical systems. At each acquisition frame each miniaturized microscope projects image data for a small section of the sample object directly onto a detector and the individual frame data are then used to form an image of the entire sample object by hardware or software manipulation.
The axial position of the array with respect to the sample object is preferably adjusted to ensure that all parts of the sample surface are imaged in a best-focus position. Thus, the detector array provides an effectively continuous coverage along the first dimension which eliminates the need for mechanical translation of the microscope in that direction, providing a highly advantageous increase in imaging speed by permitting complete coverage of the sample surface with a single scanning pass along the second dimension. Such miniaturized microscopes are capable of imaging with very high resolution. Therefore, large areas are imaged without size limitation and with the very high resolution afforded by the miniaturized microscopes.
In a similar effort to provide a solution to the challenge of imaging large areas with high magnification, U.S. Pat. No. 6,320,174 (Tafas et al.) describes a system wherein an array of optical elements is used to acquire multiple sets of checkerboard images that are then combined to form a composite image of the sample surface. The sample stage is moved in stepwise fashion in relation to the array of microscopes (so called “step-and-repeat” mode of acquisition) and the position of the sample corresponding to each data-acquisition frame is recorded. The various image tiles are then combined in some fashion to provide the object's image. The patent does not provide any teaching regarding the way such multiple sets of checkerboard images may be combined to produce a high-quality high-resolution composite image. In fact, while stitching techniques are well known and used routinely to successfully combine individual image tiles, the combination of checkerboard images presents novel and unique problems that cannot be solved simply by the application of known stitching techniques.
For example, physical differences in the structures of individual miniaturized objectives and tolerances in the precision with which the array of microscopes is assembled necessarily produce misalignments with respect to a common coordinate reference. Moreover, optical aberrations and especially distortion and chromatic aberrations, as well as spectral response and gain/offset properties, are certain to vary from microscope to microscope, thereby producing a checkerboard of images of non-uniform quality and characteristics. Therefore, the subsequent stitching by conventional means of multiple checkerboards of image tiles acquired during a scan cannot produce a high-resolution composite image that precisely and seamlessly represents the sample surface.
For instance, as illustrated in FIG. 3, assume that a conventional step-and-repeat array microscope of the type described by Tafas et al. includes only two adjacent miniaturized microscopes producing respective images 14 and 16 and that the second microscope introduces a slight rotation and offset in the image 16 acquired from the sample surface 18 with respect to the image 14 acquired by the first microscope (the dashed line represents a perfectly aligned image). In such case the acquisition of the first frame of image tiles 14,16 would produce a pattern similar to that illustrated in the figure. The acquisition of the second frame of image tiles 14′ and 16′ would produce a similarly misaligned set of images, as illustrated.
If conventional stitching procedures are used to combine the various image tiles so acquired, such as described in U.S. Pat. Nos. 5,991,461 and 6,185,315, the stitching of images 14 and 14′ will produce a seamless image of uniform quality accurately representing the corresponding section of the sample surface 12. This is because both images 14 and 14′ result from data acquired with the same miniaturized microscope and the same spectral response, gain, offset, distortion and chromatic aberrations (to the extent they have not been removed by correction) apply to both images, thereby producing a composite image of uniform quality. Inasmuch as stitching procedures exist that are capable of correcting misalignments between adjacent image tiles, a similar result could be obtained by stitching images 16 and 16′.
The combination of images acquired with different microscopes, however, could not be carried out meaningfully with conventional stitching techniques. Combining image 14′ with image 16, for example, may be possible as far as misalignments and offsets are concerned, but the combined difference could still be non-uniform with respect to spectral response, gain, offset, and distortion or chromatic aberrations (depending on the characteristics of each miniaturized microscope). Therefore, the overall composite image could represent a meaningless assembly of incompatible image tiles that are incapable of producing an integrated result (like combining apples and oranges).
Thus, the prior art does not provide a practical approach to the very desirable objective of imaging a large area with an array microscope in sequential steps to produce checkerboards of images that can later be combined in a single operation simply by aligning any pair of adjacent image tiles. Similarly, the prior art does not provide a solution to the same problem of image non-uniformity produced by the array microscope of U.S. Pat. No. 7,184,610 that is scanned linearly over a large area of the sample surface to produce image swaths that are later combined to form a composite image.
U.S. Pat. No. 5,768,443 (Michael et al.) describes a method for coordinating the fields of view of a multi-camera machine by pre-calibrating each camera for distortion to develop a distortion-correction map, applying the correction to images acquired simultaneously with each camera, and combining the corrected images to produce a composite image. While the Michel patent does not relate to microscopes, it provides a useful approach to solving the problem addressed by the present invention. This invention provides a general and efficient solution toward this end using the linear-scan array microscope of U.S. Pat. No. 7,184,610.